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ABSTRACT 

Meta-analysis is a technique for combining the 
summary statistics from previously conducted research studies to 
indicate the direction of results and provide an index of the 
magnitude of effect size. This paper focuses or ihe effect of the 
violation of the assumption of independence (that the value of any 
included statistic is in no way predictable from the value of any 
other included statistic) when combining correlation coefficients to 
determine effect size. A Monte Carlo simulation used the following 
four parameters and specified values: (1) the sample size within a 
study (20, 50, 100); (2) the number of predictors (1, 2, 3, 5); (3) 
the population intercorrelated among predictors (0 r .3 r .7); and (4) 
the population correlations between predictors and criterion (0, .3, 
•7). Path diagrams are given for each predictor case. The means, 
medians, and standard deviations of the correlation coefficients and 
the Fisher's Z transformation of the correlation coefficients for all 
population correlations and population intercorrelat ion values for 
each sample size were calculated and the data is presented in table 
format. As all the standard deviations were very close to their 
expected values, it is concluded that nonindeperdence does not affect 
the estimation of either the measures of central tendency or the 
standard deviations when the same population parameter is being 
estimated. (BS) 
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Meta-analysis is a technique for combining the summary statistics from 
previously conducted research studies. Pioneered by Gene V Glass (1976) 
meta-an. lysis gives not only an indication of the direction of the results of 
the studies, but provides an ir.dex of the magnitude of the effect as well. 
Meta-analyses are reported in terms of n.ean effect size, Is. There are two 
types of effect sizes. An experiment feet size is the mean of the experi- 
mental group minus the rrean of the control group divided by the standard 
deviation, 



ES = 




while a correlational effect size is simply a correlation coefficient, 
E5 = r. 

Meta-analysis has been further refined by Hedges (1983), who has been developing 
techniques for using effect sizes as data points and then fitting regression 
models. The focus of this paper, however, will be the use of correlation 
coefficients in meta-analyses and the effect of the violation of the assumption 
of independence in these analyses. 
I ndependence 

A necessary assumption for the results of statistical analyses to be tenable 
is independence. All inferential statistical techniques require independence of 
observations. By independence is meant that the probability of including one 
subject or data point will in no way affect the probability of including any other 
subject or data point. Another way of defining independence is to say that i;he 
value of a variable for a subject is not predictable fro;n the value of a variable 
for any other subject. 

So far independence has been defined in reference to primary studies performed 
by researchers who draw a random sample of subjects, measure the subjects on 
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variables of interest, and calculate statistics from the measured data using 
their hypothesized models. The meta-anal ys ts , on the other hand, draw a sample 
of studies usually from journal articles, record the numerous statistics 
reported in each study, and calculate a statistic based on effect sizes or a 
meta-statistic from a data set ot simple statistics. When jumping from the 
level of individual studies to combinatory techniques, studies parallel subjects 
and simple statistics parallel observations on variables. In the framework of 
combinatory methodology, then, independence means that the value of any statistic 
which is included should in no way be predictable from the value of any other 
included statistic. 

The typical study which is chosen for inclusion in a meta-analysis, however, 
will yield more than one effect size or simple statistic. When the meta-analyst 
uses all the statistics available in a particular study to calculate the mean 
effect size, the assumption of independence is violated. Landman and Dawes (1982) 
outline five ways in which the assumption of independence can be violated in meta- 
analyses. These five types of violations are as follows: 

"1) Multiple measures from the same subjects, . 

2) Measures taken at multiple points in time from'the 
same subjects, . . . 

3) Nonindependence of scores within a single outcome 
measure, . . . 

4) Nonindependence of studies within a single article, 

and 

5) Nonindependent samples across orticles" (pp. 506-507). 

Kraemer (1983) specifically provides the caveat that "only one effect size 
per study can be used to ensure independence" (p. 99) in meta-analyses. This 
means that t!,e ratio of effect sizes to studies in a meta-analysis should be 
one in order to avoid violating this assumption. However, even a cursory review 
of published meta-analyses reveals that the assumption of independence is, in 
fact, seldom met. 
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Purpose 

The purpose of this study was to determine the effect of the violation 
cf the assumption of independence on the distribution of r and the distribution 
of Fisher's Z. In this Monte Carlo simulation the following four parameters were 
used with the values specified: 

N - the sample size within a study (20, 50, 100), 

p - the number of predictors (1, 2, 3, 5), 

rho(i) - the population intercorrelation among predictors 
(0, .3, .7), 

rho(p) - the population correlation between predictors and 
criterion (0, .3, .7). 

Predictor and criterion variables were generated to conform to all possible 
combinations of the parameters specified above and then correlated. The main 
parameter of interest was rho(i), since it was the index of noni ndependence when 
it assumed a nonzero value in t^e multiole predictor cases. When only one predictor 
was used or when the intercorrelation among predictors, rho(i), equaled zero, then 
the assumption of independence was no* violated. 
Method 

In this study dependent and independent correlations were generated between 
criterion and predictor variables. The values of the parameter p, the number 
of predictors, were one, two, three, and five, and path diagrams for each case 
appear in Figures 1 through 4 respectively. In these diagrams the G variables 
are the common generating variables used along with error to form the X variables 
or predictors, which are in turn combined along with error to produce the Y or 
criterion variables. The arrows between variables indicate the relationship 
among the endogenous variables. The associated lower case letters are the 
standardized regression coefficients for path analysis. The arrows which are not 
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connected indicate exogenous variation, and those coefficients are given as well. 

The following algorith derived by Knapp and Swoyer (1967) was used to 
generate correlated vectors of numbers: 

Y = aX + f'l - a 2 Z 

where X = a vector of randomly chosen numbers from the standard normal distribution, 
Z = another vector of randomly chosen numbers from the standard normal 

distribution, and 
a = the desired correlation between X and Y. 
In the unique one predictor case, the intercorrelation among predictors 
could not be varied since only one predictor was present. Therefore, independence 
exists in this case. Here the X! vector was set equal to G, a vector of randomly 
chosen standard normal deviates, so the path coefficient between G and XI is one. 
The path coefficient between XI and Y, a, was set equal to the population correlation 
between predictors and criterion, rho(p). Since a -- rho(p), the error coeficient 
for Y was / 1 - a 2 or 71 - rho(p)2. The Y vector was then created as follows: 

Y = aXl + V'T - a 2 Z 

where Z = a vector of randomly choser, numbers from the standard normal distribution. 
The vectors for XI and Y were then correlated. 

A different procedure was used for data generation in the multiple predictor 
cases. In Figure 2, path coefficients a = b and c = d. In Figure 3, a = b = c 
and d = e = f. In Figure 4, a = b = c = d = e and f = g = h = i = j. In these 
three diagrams the correlations between any two predictors is equal to the product 
of the path coefficients connecting those two predictors with the generating variable 
or the quantity, a 2 , since all the coefficients between generating variables and 
predictors are equal. For the correlation between two predictors to equal rho(i), 
the path coefficient, a, was set equal to ?7ho~(7). Then all the X vectors were 
generated as follows: 
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*(i ) = i'a'G » )'l - aZ'i) 

"here X(i) • a vector of values for a predictor anrt i a « 

preuicior and i assumes incremental values 

for vectors from one to p, the number or predictors, 
a ■ rho(i) . the population interrelation among predictors, 
2(0 ■ a vector of randomly chosen standard nonma, deviates and , assumes 

incremental va!ues for vectors from one to p, „.e „ umber of predictors 
fhe following points concern the generation of the V vectors. First it 
should be noted that each V „ a linear combination of the n predictors plus 
error. The weight of that combination is c in Figure 2 , d ,„ Figure 3 , a „ d 
f in «9ure 4. Second, it 5hould be noted ^ ^ ^ 

reconstructed from the standardized regression coefficients in a patn diagram 
I« Figure 2. the co-relations between the two predictor, and tbe criterion can be 
reconstructed as follows: 



r yx, = c + ab d, 



r yx 2 = d + bac ' 



but since c ^ d, ,nd a = b =-/ r ho7T), the correlation between y and an y predictor, 
can be written as follows: 



yx i 



= c + P(\) c = c (l + P{\)). 



Also since r yx _ is an estimate of rho(p)> ^ ^ ^ ^ ^ 
equation so that it can be solved for c as follows: 
P{9) = c(l + p{\)) 



In Figure 3 in parallel fashion, the correlations between the three predictors 
and the criterion can be reconstructed as follows: 
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r yx 2 = e + bcf + bad, 



" yx ^ = f + cbe + cad, 



but since a = b. c , and d - e = f, the correlation between V and any 

precnctor, X(i), can be written as follows: 



r yx . = d + r (T) d + J(f) d = d(1 + 2< P(i)) 



yx . is an estate of rho(p), that value can be substituted into the 



Al3o since r is 

yx i 

equation so that it can be solved for d as follows: 

, ? (p) = d(l + 2 r <(i)) t 

d = .'(p) 

1 + 2TT1T • 

I" nwr. 4 the l„, obvious p.,.,,., exists. The correlatjons ^ 
five predictors and the criterion can be reconstructed as follows: 



the 



r 

yxj 


= f 


+ abg + ach 


+ adi 


+ aej, 


r 

yx 2 


= g 


+ baf + bch 


+ bdi 


+ bej, 


r 

yx 3 


= h 


+ caf + cbg 


+ cdi 


+ cej, 


r 


= i 


+ daf + dbg 


♦ dch 


+ dej, 


r 

yx 5 


= j 


+ eaf + ebg 


+ ech 


+ edi , 



ion 



"»»«•»«.. -»-..d...,r7Cn). andf = g . h = i=j , the 
between V and any predictor, x(i), can be written as follows: 

r yx. = f *PU)f Vt')f ♦,->(i)f * p(i)f = f(j t 4^(1)). 

*)•<" r yX( estimates rho(p) so with the appropriate substitutions the solution 
for f is as follows: 
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Av) 'fllM/d)), 

r+ 4,^(0 . 

So far in generating the Y variables in the two, three, and five predictor 
cases, the weights of the combinations, c, d, and f, respectively, have solutions. 
But in each case a weight for the error term - S needed. In the Knapp and Swoyer 
algorith, the value a 2 can be viewed as r 2 , the amount of .ariance accounted for, 
so 1 - a 2 is the amount of variance not accounted for and /l - a 2 is the weight of 
the error vector, Z. 



In the three multiple predictor cases studied here, formulas for th 2 R 2 values 
are given below: 

, 9 = c ^ + c /° 



R y l2 ' c >>yx + ^ - 2c/>(p). 



R y-123 = + d ^ yx + ^ = 3d/>(p), 



R 2 



"2 ^3 
The Y variables were generated as follows: 

Y = c(xi + x2) +/i - icTTpjz, 

Y - d(Xl + X2 + X3) +/l - 3d^"( P )Z, 

Y = f(Xl + X2 + X3 + X4 + X5) + /l - 5f^(p)Z. 

Correlations between the criterion variables and each of the predictors were then 
calculated in the multiple predictor cases 

The number of replications was chosen by solving for n p in the formula for 
the standard error o f the mean o f the correlation coefficient given below- 



, — = j n c 



r 
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The value for (T_ was arbitrarily set at .01, which was deemed sufficiently 
small for precision in this study. In this formula, /> f s the population 
correlation, rho(p), and was set equal to zero. Jhe symbol, n $ , is the sample 
size and was set equal to 20. Substituting these values into the equation 
allowed n r , toe number of replications, to assume the largest value that would 
be possible among the values for parameter, rho(p) and „ $ , ,hat. were chosen for 
this study. The solution for „ r , the numoer of replications, was 500. 

For each combination of N, p, rho(i), and rho(p) and for all r and Z 
distributions, the means, medians, and standard deviations were calculated. 



Results 



The means, medians, and standard deviations of the correlation coefficients 
for all values of rho(i), rho(p), and the number o.f predictors, p. when N=20 
appear in Table 1. The same information when N = 50 and N » loo'appears in ' 
Tables 2 and 3 respectively. 

The means, medians, and standard deviations of the Fisher's z transformation 
of the correlation coefficients for all values of rho(i), rho(p), and the 
n»ber of pr e d(ctors , „, when „ . 2p appeap fn ^ , ^ ^ .^.^ 

» h »n N = 50 and N 100 appears in Tables 5 and 6 re-pectively. 

Inspection of these tables shows that when the population correlation 

coefficient, rho(p), equals zero both the r,ean „f r and the median of r hover 

around that value and neither is consistent,, higher or lower than the other. 

However, when rho(p) assumes a nonzero value the median of r is usually larger 
than mean r. This is because r is a biased statistic and its distribution is 
negatively skewed when rho(p) is positive. This ordering of the mezn and the 
»ed lan when, rho(p) is not zero does not occur in the Fisher's 2 dutributlon. 

As N increases both the mean of r and the mean of Z are better .-.maters 
of the parameter rho(p). This follows from the Central Limit Theorem. Both 
the median of r and the median of Z tend to be better estimators of the population 



Table 1 



Means , Medians , 


and Standard 


Deviations for 


' Correlation 


fnpf fi r ipntc 


When N = 


20 
























rho(i) 














0 




.3 






.7 




P rho(p) r 


Md 

1 


SD 

r 


r Md 

r 


SD 

r 


r 


Md 

r 


SD 

r 


i a 0 


.015 


007 


t L ou 












.3 


.294 


.322 


/Ofi 
t c uu 












.7 


.690 


.706 


.126 












2 0 


.002 


.Oil 


.225 


-.004 -.007 


00 "3 


.002 


-.004 


.234 


.3 


.300 


.316 


.214 


.296 .299 


.208 


• C J 1 


.oil 




.7 


.683 


.698 


.129 


.692 .714 


■ 1 LJ 




. / 10 


.117 


3 0 


.001 


.003 


.230 


-.009 -.013 


. £.66 


.002 


-.007 


.228 


.3 


.295 


.313 


.213 


.289 .305 


.214 


.295 


.316 


.211 


.7 


b 






.686 .703 


.126 


.687 


.703 


.126 


5 0 


-.002 - 


.004 


.233 


n 08 .007 


227 


.004 


.000 


.221 




.293 


.309 


.216 


/ .320 


208 


.292 


.303 


.202 


.7 


b 






b 




.694 


.714 


.120 



..ith one predictor nonzero rho(i) values are undefined. 
b T , . 

This combination would generate data which are undefined. 
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Table 2 

gleans, Medians , jmd_ Standard C eylatlonsJorC^ Coefficient s 
When N = 50 



rho(i) 



P rho(p) 


r 


0 

Md r 


SD 
r 


r 


.3 
Md r 




r 


.7 
Md 

r 


SD 

r 


i a 0 


.001 


-.001 


.141 














.3 


.303 


.305 


.128 














.7 


.697 


.705 


.073 














2 0 


.005 


.000 


.142" 


-.001 


-,t)03 


.140 


.004 


.005 


.149 


.3 


.294 


.307 


.132 


.300 


.305 


.131 


.304 


.305 


.130 


.7 


.697 


.705 


.075 


.694 


.703 


.076 


.696 


.703 


.069 


3 0 


.002 


.001 


.139 


.007 


.003 


.145 


.001 


-.002 


.142 


.3 


?94 


.301 


.130 


.295 


.300 


.130 


.295 


.300 


.136 


.7 


b 






.696 


.703 


.075 


.694 


.700 


.076 


5 0 - 


.002 


-.001 


.143 


-.006 


-.009 


.144 


-.005 


-.007 


.141 


.3 


.299 


.303 


.129 


.300 


.305 


.129 


.295 


.300 


.128 


.7 


b 






b 






.699 


.705 


.071 



With one predictor nonzero rho(i) values are undefined. 
This combination would generate data which are undefined. 
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Table 3 



"-^^^^^ Coemcients 
When N = 100 ~ 



l a 



rho(i) 



0 



.3 



.7 



rho(p) r Md SD 7" ma m - 
Z r r __ Md r SD r Md SD 



0 


.008 


. UUJ 


1 no 










r 


r 


. 3 


.299 


.303 


091 














.7 


.698 


,701 


.053 














0 


.004 


.003 


.099 


-.008 


-.009 


.101 


.009 


.012 


.097 


.3 


.297 


.303 


.091 


.304 


.308 


.091 


.303 


.303 


.088 


.7 


.700 


.704 


.051 


.699 


.703 


.053 


.699 


703 


.048 


0 


-.005 - 


.009 


.098 


.002 


.002 


.102 


-.001 


.000 


.097 


.3 


.301 


.305 


.092 


.302 


.305 


.092 


.300 


.302 


.088 


.7 


b 






.698 


.701 


.050 


.695 


.699 


.050 


0 


-.002 - 


002 


.099 


.003 


.001 


.100 


-.003 - 


.002 


.100 


.3 


.295 . 


298 


.093 


.296 


.302 


.093 


.302 


.306 


.094 


.7 


b 






b 


























.699 


.702 


.051 



*" Uh °" e prcdictor "««•'• *o(0 values are undefined 
This combination would generate data wnich are undefined. 
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Table 4 

Means, Medians, and S tandard Deviations f ojlfishej^ljrj^^ 
of the Correlation Coeffi cient: When N = 20 



rho(i) 







0 






.3 






.7 




P rho(p) 


7 


Md 

z 


SD 
z 


Z 


Md 
z 


SD 
z 


Z 


Md 

2 


SD 
z 


l d 0 


.016 


.007 


.243 














.3 


.317 


.334 


.233 














.7 


.885 


.879 


.237 














2 0 


.002 


.011 


.238 


-.004 


-.007 


• L JJ 




- . 004 


.247 


.3 


.327 


.327 


.246 


.321 


.309 


.240 


.323 


.321 


.242 


.7 


.873 


.864 


.242 


.890 


.895 


.241 


.89 J 


.887 


.230 


3 0 


.001 


.003 


.244 


-.009 


-.013 


.246 


.002 


-.007 


.241 


.3 


,321 


.324 


.244 


.313 


.315 


.244 


.321 


.327 


.242 


.7 


b 






.879 


.874 


.242 


.880 


.073 


.241 


5 0 - 


.002 - 


.004 


.246 


.009 


.007 


.240 


.004 


-.001 


.233 


.3 


.319 


.319 


.248 


.334 


.331 


.240 


.316 


.313 


.231 


.7 


b 






b 






.891 


.895 


.229 



With one predictor nonzero rho(i) values are undefined. 
b This combination would generate data which are undefined. 



ERIC 



17 



16 



Table 5 

Means, Medians, o landard Deviations for Fis h er's Z Transformation 
of the Correlation Coefficients When N = 5Q 

rho( i ) 

0 .3 .7 

p rho(p) 7 Md SD 2 Md SD 7 Md SD 
L JL Z I r z 



5 



0 


.001 


-.001 


.144 














.3 


.319 


.315 


.144 














.7 


.876 


.877 


.144 














0 


.005 


.000 


.145 


-.001- 


-.003 


.142 


.004 


.005 


.152 


.3 


.309 


.317 


.146 


.316 


.315 


.147 


.C20 


.315 


.146 


.7 


.877 


.877 


.145 


.870 


.873 


.147 


.873 


.873 


.136 


0 


,002 


.001 


.141 


.007 


.C03 


.148 


.001 


-.002 


.145 


.3 


.309 


.310 


.146 


.310 


.310 


.145 


.311 


.309 


.152 


.7 


b 






.874 


.874 


.145 


.870 


.867 


.149 


0 


-.002 


-.001 


.146 


-.006 


-.009 


.147 


-.005 


-.007 


.144 


.3 


.315 


.313 


.145 


.316 


.315 


.145 


.310 


.310 


.143 


.7 


b 






b 






.878 


.877 


.141 



a With one predictor nonzero rho(i) values are undefined. 
b This combination would generate data which are undefined. 
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Table 6 

Means, Medians, and St an dard Deviations for F ish er's Z Transfo rma t i on 
of the Correlation Coefficients When N = 1QQ 

rho( i ) 

0 .3 .7 

p rho(p) 7 Md 2 SD Z Z Md z SD Z I Md z SD Z 



5 



0 


.008 


.005 


.110 














.3 


.311 


.313 


.101 














.7 


.870 


.869 


.102 














0 


.004 


.003 


.101 


-.008 


-.009 


.102 


.009 


.012 


.098 


.3 


.309 


.312 


.100 


.317 


.318 


.101 


.316 


.313 


.098 


.7 


.874 


.875 


.100 


.873 


.872 


.104 


.872 


.874 


.094 


0 


-.005 


-.009 


.099 


.002 


.002 


.103 


-.001 


.000 


.098 




.313 


.315 


.102 


.315 


.315 


.103 


.313 


.312 


.097 


.7 


b 






.870 


.869 


.097 


.863 


.865 


.097 


0 


-.002 


-.002 


.100 


.003 


.001 


.101 


-.003 


-.002 


.101 


.3 


.308 


.308 


.103 


.309 


.311 


.102 


.315 


.316 


.105 


.7 


b 






b 






.871 


.872 


.100 



With one predictor nonzero rho(i) values are undefined, 
'This combination would generate data which are undefined. 
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parameter, rho(p), as N increases as ,?11. Both the mean and the median are con- 
sistent estimators. It should be rerr.e-bered here that when r equals zero, Fisher's 
Z also equals zero. However, when r is .3, Z is .31; and when r is .7, Z is .867. 

Inspection of the tables shows f2t there is no discernible trend in mean r, 
mean Z, median r, and median Z over 1 e /el s of rho(i) or levels of p. This seems 
to indicate that nonindependence of f: data doe? not affect the estimation of 
the population parameter, rho(p). TVs is, of course, only for the case when the 
same parameter is being estimated by c'l the data. 

When evaluating the standard deviations they should be referenced to the 
known expected values in the cases when independence is not violated. For the r 
distribution, the standard er"or of r can be found by substituting the values 
for the parameters used in this study into the following formula: 

I 1/ ■ 

' n 

Therefore, the standard error of r when rho(p) is 0 and N is 20 is approximately 
.224. The standard error of r when m(p) is .3 and N is 20 is approximately .204. 
The standard error of r when rho(p) .7 and N is 20 is approximately .114. When 
rho(p) is 0 and N is 50 the standard i~ r0 r of r is approximately .141. When rho(p) 
is .3 and N is 50 the standard error cf r is approximately .129. When rho(p) is .7 
and N is 50 the standard deviation is approximately .072. The standard error of r 
when rho(p) is 0 and N is 100 is .1. The standard error of r when rho(p) is .3 
and N is 100 is approximately .091. Finally, the standard error of r when rho (p) is 
.7 and N is 100 is approximately .052. 

Inspection of Tables 1, 2, and 3 shows that all the standard deviations are 
close to their expected values. The largest deviation of the standard deviation 
from its expected value was .015 anc that was in an independent case. This 
deviation is of no practical concern. There is some improvement as N increases 
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because standard deviations are consistent estimators, but there are no apparent 
changes over levels of rho(i) or p. 

For the Fisher's Z distribution, the values of the standard deviations can 
be found by substituting the values for the parameter used in this study into 
the following formula: 



r 



; n - 3 . 

Therefore, the standard error of Z when N is 20 is approximately .243. The 
standard error of Z when N is 50 is appr imately .146. Finally, the standard 
error of Z when N is 100 is approximately .102. 

Again inspection of Tables 4, 5, and 6 shows f.at all the standard deviations 
are very close to their expected values. There is some improvement in the estimates 
as N increases, but there are no apparent" changes- over either levels of rho(i) or p. 

CO lCI ■jS'iOii 

The general conclusion, then, is that nonindependence does „ot affect the 
estimation of either the measures of central tendency or Me standard deviations 
for correlate coefficients and for Fisher's Z transformation of the correlation 
coefficients whei the same population paremeter is being estimated. 
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